Screening masses in neutral two-flavor color superconductor 
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I. INTRODUCTION 



Mei HuanJ^ and Igor A. ShovkovjQ 

Institut fur Theoretische Physik, J.W. Goethe- Universitat, D-60054 Frankurt/Main, Germany 

(Dated: February 2, 2008) 

The Debye and Meissner screening masses of the gluons and the photon in neutral and /3- 
equilibrated dense two-flavor quark matter are calculated. The results are presented in a general 
form that can be used in gapped as well as gapless color superconducting phases. The results for the 
magnetic screening masses indicate that the system develops a chromomagnetic instability. Possible 
consequences of the instability are discussed. 

PACS numbers: 12.38.-t, 12. 38. Aw, 12.38.Mh, 26.60.+C 

o 
o 

From the time when the quarks were predicted [3], their nature has remained rather elusive. The reason is that 
direct experimental studies of quarks are very limited. Quarks do not exist in vacuum as free particles. Under 
normal conditions, they are always confined inside hadrons. The underlying theory of strong interactions quantum 
chromodynamics (QCD) predicts that quarks should become deconfined at very high temperatures and/or very 
high densities Unfortunately, it is very difficult to achieve sufficiently high densities and/or temperatures in 

, laboratory. 

Very high temperatures existed in the early Universe during the first few microseconds of its evolution Nowadays, 
somewhat similar conditions, although at considerably smaller scales and for much shorter periods of time, are 
• repeatedly recreated in the so-called "little bangs" at the heavy ion colliders in CERN and at Brookhaven. 

Sufficiently high densities may exist in the present Universe inside central regions of compact stars. Recently, this 
possibility attracted a lot of attention when it was suggested that various color superconducting phases with rather 
^1 ' large values of gaps in their quasiparticle energy spectra could appear at densities that exist inside stars 0, IE 13 • 
If this turns out to be true, this would be of prime importance. The presence of large gaps in the energy spectra 
-■■«. ' can possibly be inferred from a detailed analysis of the observational data. This would provide a confirmation of the 
'■^ \ existence of new (quark) states of matter inside compact stars. 

In theoretical studies of dense quark matter, it should be appreciated that matter in the bulk of stars is neutral and 
/3-equilibrated. Under such conditions, the chemical potentials of different quarks should satisfy nontrivial relations. 
These, in turn, affect the pairing dynamics between quarks which is reflected in a specific choice of the ground state of 
matter. For example, it was argued in Ref. that a mixture of the two-flavor color superconducting (2SC) phase and 
normal strange quarks is less favorable than the color-flavor-locked (CFL) phase after the charge neutrality condition 
is enforced. A similar conclusion was also reached in Ref. 

Assuming that the constituent medium modified mass of the strange quark is large (i.e., larger than the correspond- 
ed ■ ing strange quark chemical potential) , it was shown recently that neutral two-flavor quark matter in /3-equilibrium can 
have another rather unusual ground state called the gapless two-flavor color superconductor (g2SC) [lj. While the 
symmetry in the g2SC ground state is the same as that in the conventional 2SC phase, the spectrum of the fermionic 
quasiparticles is different. In particular, two out of four gaped quasiparticles of the conventional 2SC phase become 
gapless in the g2SC phase. In addition, the number densities of the pairing quarks in the g2SC phase are not equal at 
zero temperature For example, the density of red (green) up quarks is different from the density of green (red) 
down quarks. 

The existence of gapless color superconducting phases was confirmed in Refs. [l2l Il3|. and generalized to nonzero 
temperatures in Refs. ll^Llla . In addition, it was also shown that a gapless CFL (gCFL) phase could appear in neutral 
strange quark matter jl6L Il7|. At nonzero temperature, the gCFL phase and several other new phases (e.g., the so- 
called dSC and uSC phases) were studied in Refs. |l8l Il9|. If the surface tension is sufficiently small, as suggested 
in Ref. [2(j , the mixed phase composed of the 2SC phase and the normal quark phase will be more favored |2lj| . It 



'Electronic address: huang@th.physik.uni-frankfurt.de on leave of absence from Physics Department, Tsinghua University, Beijing 100084, 
China 

^Electronic address: shovkovy@th.physik.uni-frankfurt.de on leave of absence from Bogolyubov Institute for Theoretical Physics, 03143, 
Kiev, Ukraine 



2 



was also suggested that a non-relativistic analogue of gapless superconducting phases could appear in a trapped gas 
of cold fermionic atoms [22I I2M p3. J25ll . (Note that an alternative ground state for the atomic system, similar to the 
quark mixed phases in Refs. |2fil2lLl2fij ]. was proposed in Ref. |27j.) 

While the basic properties of gapless color superconducting phases have been established in Refs. 0, 0, 0, 
IT5IIT6L Ell ll8L ITgj . there is probably much more to be learned about these phases in the near future. In this paper, we 
give a detailed derivation of the gluon and photon screening masses in neutral dense quark matter (the results were 
briefly presented in Ref. H^). We consider the general case of two-flavor quark matter, covering both the gapped and 
the gapless 2SC phases. 

The gluon screening properties in the case of the ideal 2SC phase (i.e., without a mismatch between the Fermi 
momenta of different quarks) were considered in detail in Refs. [2|||3(|. Some general features of the gluon screening 
in the g2SC phase were conjectured in Refs. 0, without performing the calculation. As we shall see below, the 
actual results for the Debye screening masses are in general agreement with the conjecture in Refs. [Til l3lj . The 
Meissner (magnetic) screening properties, however, are very different psj . The calculations in this paper indicate a 
chromomagnctic instability in neutral dense quark matter for a range of parameters in the model. As we shall argue 
in Sec. IVIIII this instability may lead to a gluon condensation in dense quark matter. 

This paper is organized as follows. The linear response theory is briefly reviewed in Sec.[n] After that, in Sec. IIHI 
we introduce the model and set up the main notation. There, we also present the general expression for the quark 
propagator in the color superconducting ground state of neutral dense quark matter. In Sec. IIVI we briefly discuss 
the general expression for the polarization tensor in dense quark matter. In Sec.0 we study the polarization tensor 
H AB f° r A,B = 1,2,3 and derive the corresponding expressions for the Debye and Meissner screening masses. We 
show that, in accordance with the symmetry breaking pattern, there is no Meissner effect (i.e., no Higgs mechanism) 
in this sector of the gauge theory. The polarization tensor H AB for A, B — 8, 9 (i.e., the 8th gluon and the photon) is 
discussed in Sec. I VII There we derive the Debye and Meissner screening masses and briefly discuss their properties. 
Also, the mixing between the 8th gluon and the photon is discussed. In Sec. IVIII we study the polarization tensor 
Tl AB for j4, B = 4, 5, 6, 7. The results for the Debye and Meissner screening masses are presented. As we show, the 
Debye screening mass is given by a rather simple expression that naturally interpolates between the limits of the 
normal phase and the ideal 2SC phase. The Meissner mass, on the other hand, has an unexpected property. Its value 
squared is negative in a range of parameters, indicating a chromomagnetic instability in dense quark matter. The 
discussion of the main results is given in Sec. IVIIII Our findings are summarized in Sec. 11X1 Several appendices at 
the end contain useful formulas and some details of the calculation. 



II. LINEAR RESPONSE THEORY AND POLARIZATION TENSOR 



In order to present a self-contained discussion of the screening properties of dense quark matter, we start this paper 
with a brief introduction into the linear response theory and a discussion of the physical meaning of the polarization 
tensor in a gauge theory. The advanced reader, therefore, may skip this section and go directly to Sec. 11111 

The response of matter to an external perturbation is the main source of knowledge about properties of matter. 
The linear response theory is the simplest framework that is often used to calculate such a response. In application to 
quark matter, for example, one studies a response of the system to an external source 3 A (x). The source is coupled 
to the quantum gauge field. The corresponding interaction part of the action reads 

Sj = J d 4 xA A ^(x)3 A (x) 

ee ijd*xj d%A A ^{x) (D- 1 )^ {x - y)A B '"(y), (1) 

where A B '"(y) is the classical field associated with the external source J A (x), and (Z) -1 )^ is the inverse free gluon 
propagator. Because of the presence of the external source, the expectation value of the gauge field becomes nonzero. 
In the linear response theory, it is given by the Kubo's formula [32jj, 

(A A (x)) = -ij d^yV AB (x - y)r- B (y), (2) 

where T> AB (x — y) is the retarded gluon propagator. In momentum space, this relation takes the following form: 



(A A (P)) = -W AB (P)3 V ' B (P), 



(3) 
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where P M = (p , p) is the energy-momentum four-vector. By making use of this result, it is instructive to derive an 
expression for the induced current. It is given by 



J, 



A,ind 



, AB 



U AB (P)(A B ^(P)), 



(4) 



where Il^f (P) is the gluon self-energy (or the gluon polarization tensor). By definition, this is the one-particle 
irreducible part of the gluon two-point function. The structure of this function is constrained by the gauge symmetry. 
To see this we consider the Slavnov- Taylor identity for the full gluon propagator ■ The explicit form of this identity 
depends on a specific gauge fixing. In the covariant gauge, for example, it reads 



P»P"iV A ?(P) 



P»P v iD AB (P) 



1 

A ' 



(5) 



where A is the gauge fixing parameter. In vacuum, where Lorentz symmetry is not broken, this relation implies that 
the gluon self-energy is transverse, i.e., P^U AB (P) — 0. Because of this constraint, the tensor structure of U. AB (P) 
in vacuum is fixed unambiguously, 



n^(p) = n^(p 



AB i 



P^Py 
P 2 



(6) 



where the metric tensor is defined as g M „ = diag(l, — 1, — 1, — 1). At nonzero temperatures and/or densities, the 
Lorentz symmetry is broken down to its subgroup of spatial rotations 5*0(3). Then, the tensor structure of the gluon 
self-energy can have a more general form, 



n 



AB.iiv 



(P) 



L 



AB 



H AB + u»u v K AB 



-AP 



(7) 



where = (1,0,0,0) is a time-like four-vector that specifies the rest frame of the quark system and p'' = (0,p) is 
the momentum three- vector with the absolute value p = |p|. The component functions H AB , K AB , L AB , and M AB 
are functions of po and p. Now, the Slavnov- Taylor identity in the covariant gauge takes the form 



KL + p\K - p 2 L + M(M - 2p p) = 0, 



(8) 



(for simplicity, the superscripts "AB" were omitted here). It is easy to check that this is less restrictive than the 
transversality condition, P fJ Tl llu (P) — 0, required in vacuum. Indeed, the transversality is equivalent to the following 
set of two relations between the component functions: 



L 



4k 



Po 



K. 



(9a) 
(9b) 



While these are sufficient conditions to fulfil the Slavnov- Taylor identity in Eq. ©, they are not the necessary 
conditions in a non-Abelian gauge theory when the Lorentz symmetry is broken. (It should be emphasized, however, 
that these are the necessary conditions in Abelian gauge theories at nonzero temperatures and/or densities 

In this paper, we study the polarization tensor Il^f (P) in the case of dense quark matter which permits color 
superconductivity. In particular, we discuss how the structure of the polarization tensor is affected by the (gapless) 
color superconductivity. 

It is usually said that superconductivity is a result of a gauge symmetry breaking. This common misleading 
statement may suggest that the polarization tensor U AB (P) does not need to satisfy the Slavnov- Taylor identity 
In fact, this is not the case because a local (gauge) symmetry can never be truly broken |3^. 

In practice, when doing specific calculations in gauge theories, one always breaks the local symmetry by a gauge 
fixing. As a result, it is only a global symmetry that may remain unbroken after the gauge choice is made. For example, 
in the covariant gauge which we discussed above, the global color symmetry of the QCD action is left unbroken. Then, 
in a color superconducting phase of quark matter, this global symmetry is broken. In such a description, the Goldstone 
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theorem requires the appearance of an appropriate number of the Nambu-Goldstone bosons (cf., collective excitations 
in Ref. H^]). Obviously, the appearance of these additional degrees of freedom is an artifact of the gauge fixing. 
Nevertheless, their inclusion in the analysis is important in order to insure that the general requirements of the gauge 
symmetry, such as the Slavnov- Taylor identity (JSJ, are fulfilled [3(J. Having said this, one should appreciate that 
the Nambu-Goldstone bosons in question are not the physical degrees of freedom. This conclusion is easy to reach 
by noticing that there exist a gauge, namely the so-called unitary gauge, in which these bosons can be completely 
eliminated. In a way, their role is similar to the role of the Faddeev-Popov ghosts [31J. While both types of fields are 
unphysical, they are necessary for a consistent description of a gauge theory. 

Now let us further discuss the physical meaning of the polarization tensor. From a relation similar to that in Eq. Q), 
it is clear that, in an Abelian theory such as QED, this tensor is directly related to an observable quantity, namely 
to the induced current. The corresponding current is a gauge invariant quantity in an Abelian theory. In contrast, 
the induced current in a non-Abelian theory is not a gauge invariant quantity. Then, the physical meaning of the 
polarization tensor is not so clear. In spite of this difficulty, we shall use the same interpretation of the polarization 
tensor in quark matter as in an Abelian theory. 

In this paper, we study static large-distance, electric and magnetic, screening properties of quark matter. These 
describe the response of the system to a static perturbation from color/electric charges and currents. The static limit 
means that po = 0. In this case, the only nontrivial components of the polarization tensor will be H(p) and K(p) 
which depend only on p. Note that 

n O0 (0,p) = K(p), (10a) 

n y (o, P ) = ^ gij + E^i\H(p). (10b) 

Let us denote the values of the two nontrivial component functions in the limit p — > (large distances) as follows: 

m 2 D = - lim K(p), (11a) 

p^O 

m 2 M = -limtf(p). (lib) 

p— >o 

The quantities mo and tum are the Debye and Meissner screening masses, respectively. It can be shown, see for 
example Ref. [12) GUI, that the quantity raj determines the large-distance behavior of the screened potential created 
by a static color/electric charge, i.e., V[r) ~ exp(— m^r). By making use of the analogy with solid state physics 
systems, we say that a system is a metal when mjj is nonzero, and it is an insulator when is zero. The quantity 
mju, when nonzero, determines the large-distance fall-off of the (chromo-)magnetic field inside a color superconductor, 
i.e., B(r) ~ exp(— Tom?*)- Obviously, nonzero tum is an indication of the Meissner effect. In the normal phase, on the 
other hand, the value of uim is vanishing. 



III. QUARK PROPAGATOR 



In this paper, we continue the study of dense two-flavor quark matter constrained by the conditions of the charge 
neutrality and the /3-equilibrium 0, [jjj El El El El E E El El E EH . The use of phenomenological 
Nambu-Jona-Lasinio (NJL) type models has proved to be very convenient in such studies. The NJL model can 
be thought of as a low energy theory of QCD in which (massive) gluons, as independent degrees of freedom, are 
integrated out. The gluons could be reintroduced back by gauging the color symmetry in the NJL model, providing 
a semirigorous framework for studying the effect of the Cooper pairing on the physical properties of gluons. 

In order to study the gluon screening properties in dense quark matter, we need to know the quark coupling to the 
gauge fields. This is determined by the quadratic part of the quark Lagrangian density 



£( 2 ) = $ (jyfy - m + A70 + g^A^Ta + erfAnQ) V>, 



(12) 



where T a and Q are the generators of SU(3) C and t/(l) cm gauge groups. The coupling constants of the strong 
interactions and of the electromagnetism are denoted by g and e, respectively. The up and down quark masses are 
assumed to be the same (m u = rtid = m). The quark spinor field has the following color-flavor structure: 



( Ipur 

Ipub 
Ipdr 

V tpdb 



J 



(13) 
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where i S (it, d) is the flavor index and a S (V, 5, b) is the color index. 

In /^-equilibrium, the matrix of chemical potentials in the color-flavor space jx is given in terms of the quark chemical 
potential \i (note that \xb = 3/z is the baryon chemical potential), the chemical potential for the electrical charge /i e 
and the color chemical potential /is, 

tif = Wij - HeQ l3 W P + -^M8%(T 8 )^. (14) 

In QCD the color chemical potential /is comes as a result of a nonzero expectation value of the 8th gluon field induced 
by the Cooper pairing |42| . Its absolute value is small because it is suppressed parametrically by the quark chemical 
potential, /ig ~ A 2 //i. 

The explicit expressions for nontrivial elements of matrix p, read 

2 1 , ^ 

(15a) 

(15b) 

(15c) 

(15d) 

To study color superconducting phases, it is convenient to introduce the following 8iV c .ZV/-component Nambu-Gorkov 
spinors: 

* = (V^c) , 9=(* r ), (16) 



f-L ur 


— f-^ug — 




3 Me + 


3M8 


f^dr 


= Hdg = 


n + 


1 

3 Me + 


^8 




Hub = 




2 


2 

^8 




Hdb = 




1 


2 

3^8 



where ipc — Cip T is the charge-conjugate spinor, and C = Z7 2 7° is the charge-conjugation matrix. In this basis, the 
quadratic part of the quark Lagrangian density becomes 

£ (2) = f S^ 1 * + f A^f^tf, (17) 

where the explicit form of vertex is 

f diag(«? 7 ^T A ,-g 7 ^Tj) for A = a=l,...,8, 
f^diag(r A ,r A W (18) 
[ diag(e7^Q,-e7^Q) for A = 9 (photon). 

In momentum space, the inverse free quark propagator Sq 1 reads 

The explicit form for the diagonal elements is 

+ 7° [(fco - E%)At + (fc + <)A* ] 
+ 7° [(fco - ^ b )A+ + (fc + E±)K- k ) 

+ 7° [(fco - i?J b )A+ + (fco + E^)A-} (20) 

with the notation Ef a = ± fj,i a and E^ = \/k 2 + m 2 . The four projectors (with / = 1, ... ,4) in the 6- 
dimcnsional color-flavor space are defined as follows: 

= (S a0 ~ S ab S^ b )5 iu 5 ju , (21a) 

= (S a0 - S ab S 0b )6 ld 6 jd , (21b) 

= 6 ab SP% u 5 JU , (21c) 

= S ab 6^ b 6 u S jd . (21d) 





a/3 






W 




a/3 




y 


7?(3) s 


a/3 




« 




a/3 
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It is not difficult to see that P^ projects out the red up and the green up quarks, and P^ projects out the red 
down and the green down quarks. The projectors P^ and P^ project out the blue up and the blue down quarks, 
respectively. In Eq. if^Hjl . we also introduced the energy projectors, 



o7'k- 



These projectors satisfy the following relations |43j : 



E k 



Afe i 



where 



Afc 



o 7 ' k - m 



(22) 



(23a) 
(23b) 



(24) 



is an alternative set of energy projectors. In the chiral limit, the two sets of projectors in Eqs. i|22|) and 124fl coincide. 
The full quark propagator in a color superconducting phase takes the following form: 



x_f[G+(K)] 1 A- 



A+ 



[Go(K)\ 



with 



A" = -ieVfA, 



A 



+ = 7 ° 



A~) 7° = -ie b ej 5 A* 



(25) 



(26a) 
(26b) 



where e b is the antisymmetric tensor in the color subspace spanned by the red and green colors, while e is the 
antisymmetric tensor in the flavor space. The value of the gap parameter A is determined from an appropriate gap 
equation, while the values of the chemical potentials /i e and fig are determined from charge neutrality conditions 



lOt IllL [13J, [1J, |39j, l40t |41| . The explicit form of the gap equation and the neutrality conditions is not important 
for the purposes of this paper. 
From Eq. 1(25(1 . we obtain 



where 



(27) 



G ± 



with the free quark propagators 



Gn 



+ 7 U 



fco + E ur 

At 



(Gty 1 -AT G TA± 
-G^A ± G ± , 



a; 



h - E^r 
AT 



ko + E± ko-E* 



p(3) 



Afe 



ko + E% 
At 



E 



obtained from Eq. 1)20(1 . By making use of the following relations: 



eP^e = 


~P (2 \ 




= 


eP^e = 


-P {1 \ 




= _p(2 


eP^e = 






= o, 


eP^e = 


_p(3) j 


e b V (<t) e b 


-o, 



a; 



fcn - El 



dg 



a; 



ko - E* 



p(2) 



(28a) 
(28b) 



(29) 



(30a) 
(30b) 
(30c) 
(30d) 
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one can derive an explicit form of the Nambu-Gorkov components of the full propagator, 

4 

G ± = J]G^ (/) , (31a) 
i=i 

S± = Ef 2 e b V^eV^ +Ef 1 e b V^eV (1 l (31b) 
The explicit form of the functions Gf and S^j reads 



Gf = fc °'^ +2 7°A+ + k ° + El9 2 7°A fc -, (32a) 



± = k E± o A + + fc " + ^ /A-, (32b) 



Gf = r ^7°A+ + -^ w 7°A^, (32c) 
fc o + E bu k - E+ u 



Gf = r ^7°A+ + -^ W 7°Afe, (32d) 
fc + E bd k - 



and 



3± = -iA ( -2 7 5 A+ + 27 5 Ar; ) , (33a) 



—21 



7 5 A^ + A -a 7 5 A,T 1 , (33b) 



where the following notation was used: 



Et = E k ±p, (34a) 



E 



A,k 



J{E±)* + A*, (34b) 

_ _ AW + Mrfg Hug + Hdr Me . M8 , . n 

- = ^^ = ^-T + y ( 34c ) 

r _ Hdg — AW _ fJ-dr ~ AW _ Me (34rn 

^ ~ 2 2 2 ' 1 7 

In the following sections, we use the full quark propagator in Eq. (|27Jl to construct the polarization tensor for the 
gauge fields. 



IV. POLARIZATION TENSOR IN DENSE QUARK MATTER 

In dense quark matter, screening effects play a very important role at length scales larger than the average distance 
between quarks. In the normal phase, for example, the main effects are the Debye screening and the Landau damping. 
These are the properties that can be extracted from the behavior of the polarization tensor. The polarization tensor 
in dense matter is given by the so-called hard dense loop (HDL) approximation 0, ^IJ . This approximation results 
from taking into account only the dominant one-loop quark contribution in which the internal quark momenta are 
hard (i.e., typical momenta are of order [i). The density of quark states with hard momenta is proportional to (i.e., 
the density of states at the Fermi surface). Because of this large density of states, the quark HDL contribution is large 
compared to the contributions from the gluon and the ghost loops. Therefore, the gluon and the ghost contributions 
are not included in the HDL approximation. 

In the 2SC/g2SC phase of dense quark matter, the polarization tensor is given approximately by the following 
one-loop expression [2{| H3 : 

Kb(P) = \t, E Tl Uc,f,NG \t%S{K)t» B S{K - P)\ , (35) 
z v K 
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where the trace runs over the Dirac, color, flavor, and Nambu-Gokov indices. The gluon part (A, B = 1, . 
polarization tensor reduces to the standard HDL result in the normal phase (A = 0), 



of this 



4q ^ 2 $ ab. 



9 9 



Po 
p 

Po 

p 



-u v )Q 




where a s = g 2 /4w, and 



Q(x) = - 





f + X 








- f 




1-sc 





K £ + x — ie £ — a; — is 
In the static limit (po = 0), we obtain Q(0) — —I, and the polarization tensor becomes 



n 



HDL 



(o,p) 



(36) 



(37) 



(38) 



This gluon polarization tensor describes the static screening of quark color charges at large distances in the normal 
phase of dense quark matter. By comparing with Eqs. (|10|l and we derive the corresponding expression for the 
Debye screening mass, 



Aa s ^i 2 



(39) 



As it should be, the Meissner screening mass is zero in the normal phase. 



V. GLUONS WITH A = 1,2,3 

In this section, we start with the screening properties of the A = 1,2,3 gluons. These are the gluons of the unbroken 
SU(2) C subgroup which couple only to the red and green quarks. The corresponding expression for the polarization 
tensor is diagonal, Ii A u B (P) = 8ab^-\\{P)- After performing the traces over the color, flavor and Nambu-Gorkov 
indices, we arrive at 



(2^)3 



Tr D [^G+(K)rG+(K')+rG^(K)^G^(K') 



+ 7 "G+ (K)j»G+ (JO + 7 M G 2 - (K)YG^ (K') 



(40) 



where T is the temperature and K' = K — P. Here we use the imaginary time formalism, and the energy integration 
is replaced by the sum over the fermionic Matsubara frequencies w n = irT(2n + 1). The explicit expressions for the 
components Gf and Efj of the quark propagator are given in Eqs. (|32(l and (|33|) . After the summation over the 
Matsubara frequencies, we obtain the result in the following form: 



n£(p) = 



d 3 k 



ii 



(2tt) 3 

(C?_ + Cf_) T?_ + (C n + + C 2 _\) T»1 



+ (C* n _ + C 2 _ 2 _) T 



-(JjV 



12 



c 



21 



fiiy 



12 



(cl 2 _ + cf_)u^_ + {ch 2 + 



c 2 }^)u^_ 
c 2 _\)u^ + 



In this expression, we introduced the following notation for the two types of Dirac traces: 

TZ, = TrDbVA^VA^], 



(41) 

(42a) 
(42b) 
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with ei,e2 = ±. To leading order in 1/fi, the results for these traces are given in Eqs. (|A1|) - (|A4|) in Appendix lAl 
The expressions for the coefficient functions C± ± at zero and nonzero temperatures are given in Appendix \B II 
We write the integral over the three-momentum in Eq. (|4f \ as 



**-iZ>*L*k i- (43) 



where <fi is the polar angle and £ is the cosine of the angle between the three- momenta k and p. After performing the 
integral over the polar angle </>, see Eqs. (|A5|I and (|A6|) . the corresponding traces in the integrand can be replaced by 
the following angular averaged expressions: 

rr -> 2nV T 2^ P P -(\-?)W v -u*u v )-(\-3?)?-%-, (44a) 

p p z 

T£Z -» -(l + ^W ~u^u u ) + (l-3e)^-, (44b) 

- -(l+ac^-^ + a-sc 2 )^, (44c) 



p 

V 2 



Kt -2uV-(l-e 2 )(.9 A " y -«V)-(l-3^ 2 ) ± ^-T2e^ (44d) 



Now we would like to note that there are two different types of coefficient functions in Eq. I|41|l . The coefficients 
C±±, C±±, Cj?^ and C^V originate from particle- hole and antiparticle-antiparticle loops. In the leading order 
approximation, we drop the antiparticle-antiparticle contributions to the polarization tensor. These are suppressed 
by an inverse power of the quark chemical potential. In addition, in the static long wavelength limit (p = and 
p — > 0), the particle- hole contributions simplify considerably. The approximate expressions read 



A 2 (E7 J 2 + (E k ) 2 h E7 

C X A - -——[\-B(-E^ k + &»)]-8(-E^ k + 5») K ~ ~~ ~ , (45a) 

4( £; A, fc ) ^(E Ak y 



-,22 A 2 (^A.fc) 2 + Cgfc ) 2 + 2E A,k E k 



Here we neglected the corrections due to nonzero quark masses. This is justified if the shift of the quark Fermi 
momenta due to such masses is small, i.e., m 2 / ' \x <C S[i. We also used the following relations valid when p — > 0: 

k' = |k - p| ~ A; - p£, (46a) 



k 



K * E k ~ ^K, (46b) 
E tk> * Ett-llrS-PZ, (46c) 



E k E 



A,k 



-EX k , + Sri ~ 6(-EX k + S^) + S(-E A k + 5ri^r-^Pi- (46d) 

&k E, 



A,k 

As is easy to check, the terms with the 9- and <5-functions in Eq. I|45|l contribute only in the g2SC phase when A < S/i 
(this also includes the normal phase as a limiting case with A = 0). 

Now, the coefficients C^, C^ T , C±± and C±± are made of particle- antiparticle loops only. Their contributions 
to the polarization tensor are not negligible although they are formally suppressed by an inverse power of the quark 
chemical potential. In fact, the corresponding contributions are ultraviolet divergent. The divergences are removed 
by subtractions of the corresponding vacuum terms. One can also check that a nonzero gap parameter (A -C fJ>) 
leads only to a small correction to these particle-antiparticle contributions. Thus, we approximate the corresponding 
coefficient functions as follows: 

r 11 ~ r 22 ~ \k-p.\ + k-ji l 

C± t " c ^--\k-fi\(k + f, + \k-f,\) + k' (47a) 
c ±± = c ±± - °- ( 47b ) 
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While the traces in Eq. (|44|) depend on the angular coordinate £, the approximate coefficient functions C±± in 
Eqs. (|45|l and l|47|) do not. Therefore, the corresponding angular integration in Eq. I|41|l can be easily performed, 



-l 
1 



4// - -{jgT -uV), 
o 



8 



-4«V--(/"- ) iV). 
3 



(48a) 
(48b) 
(48c) 
(48d) 



The results on the right hand side are independent of the momentum p. Therefore, in order to derive the final 
expression for the polarization tensor, we need to calculate only the following momentum integrals: 



dkk 2 C. 



2^11,22 



±± 



dkk C_[_ -p 



dkk 



dkPC^ 1 



1 -2 



1 - 



5fj,9(dp,- A) 
v 7 ^ - A 2 

6fi9(diJ,-A) 
VV 2 - A 2 



2^ 



0. 



(49a) 

(49b) 

(49c) 
(49d) 



The details of the calculation are given in Appendix IC II Note that, in the 2SC phase when 811 < A, the contributions 
from the normal (C±± ) and abnormal (C-j-i ) quark loops are equal, see Eqs. (|49a|l and l|49b|) . The additional 
contributions in the g2SC phase have equal absolute values but differ in sign. As for the particle-antiparticle quark 
loop, only the normal (C^ 22 ) contribution is nonvanishing. 

By substituting the results in Eqs. (|48|l and (|49|) into Eq. (|41|l . we derive the following expression for the polarization 
tensor IP^: 



E£(0) 



4a s /x 2 



dkk 2 [3 (C*|V - C| 2 _) - (ClV + 2C| X _ + C 12 _) - u^n")] 



VSfi 2 - A 2 



0(5/* - A)u"u". 



(50) 



Only the 00-component of this polarization tensor is nonzero. This component determines the Debye screening mass, 

4a s fl 2 5/jl 



m D.ll 



ng(o) 



-0(5jj, - A). 



(51) 



In the 2SC phase (A > <5/i), where there are no gapless excitations charged with respect to the unbroken SU(2) C 
subgroup, this screening mass is identically zero [2!j. In contrast, it is nonzero in the g2SC phase (A < 5fi). In fact, 
its value is proportional to the density of the gapless quasiparticle states. 

As for the Meissner screening mass, its value vanishes in the 2SC phase as well as in the g2SC phase, 



l M.ll 



2 P ^o 



lim gij + 



PzPj 



n£(o )P )~o. 



(52) 



VI. 8TH GLUON, PHOTON AND THE GLUON-PHOTON MIXING 

In the 2SC/g2SC phase, the diquark condensate breaks the gauge symmetry SU (3) c x U (l) em down to the SU (2) c x 
J7(l) em subgroup. The structure of the condensate in the ground is such that the 8th gluon and the photon mix. The 
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new medium fields are given by the following linear combinations of the vacuum fields |46( : 

Al= Al cos ^ + Al simp, 
A-Z = A l cos ip - Al sin if, 



(53a) 
(53b) 



where the mixing angle <p is determined from the structure of the condensate by using simple group-theoretical 
arguments, 



COS if 

sin^ 



3a, 



3a, 



3a, 



(54a) 
(54b) 



The generator of the medium (low-energy) electromagnetism U(l) em is defined as follows: Q = Q — ^Tg. 

The 8th gluon can probe the Cooper-paired red and green quarks, as well as the unpaired blue quarks. After the 
traces over the color, the flavor and the Nambu-Gorkov indices are performed, the polarization tensor for the 8th 
gluon can be expressed as 



Tr D [rG+ (K) 7 »G+ (K') + 7 "G7 (K)^G^ (K') 



(55a) 



4 ^ J (2tt) 3 
^G+ {K)YG+ {K') + ^ G 2 - {K)YG7 2 (K') 

g 2 T ^ f d 3 k 



(27T)' 



f Tr D [^G+(K)YGt(K')+rGs(K)YGz(K') 



(55b) 



(55c) 



Note that the contribution ITgg of paired quarks, up to a sign in front of the contribution from the abnormal quark 
loops, is the same as the expression for in Eq. 1)40(1 . Therefore, by following the same steps as in the previous 
section, we easily derive the result 



fi&(0) c ^ |dfcfc 2 [3(C|V 



Ci 2 _) u»u 



n 

++ 



4a s /i 



4a s ^ 2 
3tt 



1 - 



2C\ 



A 2 



(56) 



[c.f. Eq. I|5()jl]. The unpaired blue quarks give the standard normal phase HDL contribution Ilgg b (P) = Ilyp L (P), 
see Eq. ^E$. In the static (po = 0) long- wavelength (p — > 0) limit, this leads to the following result: 



iC 6 (o) 



4a 



(57) 



Thus, the final result for the polarization tensor of the 8th gluon reads 



n^(o) 



4a s /i 2 



4a s /i 2 
9vr 



5p,6(8pL - A 
TP - A 2 



(g^ - 



(58) 



Because of the symmetry considerations, one should expect a nontrivial gluon-photon mixing in the ground state. So, 
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we proceed to the calculation of the photon polarization tensor. The general expression for this tensor is 



in^(p) + ln^ 6 (p), 



(59a) 



^(n - ^E 



Tr D {4 [rG+(K)^G+(K') + ^ (K)^G^ (K')] 



+ {K)i'G+ (if') + ^G- 2 (A-)7"G?3 (if') 

+ 2 [r Sr 2 (if b^i ') + 7 M S+ (if )7 y S 2 - 1 (X')] 
+ 2 [ 7 ^2 1 (A')7 , 'S+ (if') + 7^+^)7^^')] } , 

r Tr D {4 [rGi{K)^Gi(K')+rG^{K)^G^(K')\ 



(59b) 



9 ^ 



(2tt) 3 



+ 7"G+(ir)7"G+(^') +7^4(^)7^4-^')} 



(59c) 

Let us first start with the contribution of the unpaired blue quarks Hgg b (P). This is proportional to the standard 
HDL result, Ilgg b (P) — (10a/9a s )n^ [ I jl )L (P) where a = e 2 /47r is the fine structure constant and n^ L (P) is given in 
Eq. i|36|) . At p = and p — > 0, we arrive at 



n: 



40a 



(0) = -ip/iW 



(60) 



"> feW 9tt 

To calculate the contribution of the paired quarks Ilgg (P), we use the same approach as in the previous section. After 
performing the Matsubara frequency summation, we obtain the following representation: 



n&(P) = 



d 3 k 

(2tt) 3 



[(4C|V + G 22 + ) T£l + (4C n _ + C 22 _) T!t v _ 



(4C£_ + C 22 _) T» v _ + (4C. 



2(G| 2 + + G 21 + 



Ul\ + 2 (C. 



11 

-+ 
12 



22 



g 2 



+ 2(C 1 + 2 _+Cl 1 _)U^_+2(C 12 + + C 21 + )U^ + ]. (61) 

By substituting the results in Eqs. (|48|l and 149|) into Eq. (|61|l . we derive the following expression for the polarization 
tensor Ilgg in the static long- wavelength limit: 



n^(0) ~ 1^ [ dkk 2 [3 (5Cl\ - 4C\ 2 _) u»u v - (5C\\ + + 4G| 2 _) (g^ - u»u v )] 

Z {7T J 



8a/i 2 
9tt 



9Sfid(S^L — A) 
VV - A 2 



8a/2 2 
27tt 



5/i 6>(5/z - A) 



VS^ - A 2 



By combining the results in Eqs. I|60|) and (|62p. we obtain the expression for the photon polarization tensor 



n^(o) 



8a/2 2 
3tt 



1 



3Sfi8(5fi - A) 
17$ 



4a/i 
27tt 



1 - 



5 0(S^j, — A) 

71 



A 2 



(62) 



(63) 



Now, we calculate the gluon-photon mixed components of the polarization tensor. The corresponding expression is 
given by 



1™ 



1, 



iC(P) = n^(p) = -n^(p) + -n^,(P) 



n^(P) 



2 "89 
d 3 k 



(64a) 



E / ^3Tr D {2[ 7 ^+(Jf)fG+(if') + 7"Gi(if)7"Gr(if')] 



eg? 1 

3V3 • ^ 

7"G+(if) 7 "G+(if') - 7^ (if h^" (if') 
7^r 2 (K)7-H+(X') + 7 MS+ (Jf)7"Sa 1 (Jf') 
2 [7^-^)7^+ (if') + 7 ^+(WSr 2 (iO] } , 



n^,(P) 



d 3 k 

(2^)3 



Tr D {-2 [rGt{K)^Gt(K') +7^3 (if h'Gj (#')] 



7 "G+ (if )7"G+ (if') + 7 ^G 4 - (if ) 7 ^G 4 - (if ') } 



(64b) 



(64c) 
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The contribution of the unpaired blue quarks is proportional to the HDL expression, 



n£,(P) 



:! V :!a; n ™ L(P) ' 



(65) 



with n^p L (P) defined in Eq. H36|) . At po = and p — > 0, we obtain 



n^( Q ) = V^inW 



3V3tt 



(66) 



To calculate the contribution of the paired quarks, Ilgg , we first perform the Matsubara frequency summation. The 
result is 



n£(P) 



3V3 J (2tt) 



d 3 k 



[(2C^ + - C 22 + ) T£ + (2C n _ - C 2 _ 2 _) 



(2Cl 1 _ - Cf_) T» v _ + (2C 1 _\ - c 2 \) rr+ 
(Cj 2 + - 2C\\) U%\ + (C 12 _ - 2C 2 }_) 
{C\ 2 _ - 2Cf_) + {C 12 + - 2C 2 _\) W\\ 



(67) 

By substituting the results in Eqs. (|48|l and (|49|l into Eq. I|67|) . we get the following expression for the polarization 
tensor Ilgg in the static long- wavelength limit: 



n$(o) 



^ | dkk 2 [3 (C|V + C| 2 _) uV (Cf + + 2Cf_ Cf_) (iT - 



3\/37r 



8^/ao! s /i 2 



1 - 



5[i6(5(j, - A) 
- A 2 



(68) 



[c.f. Eq. (|56p]. By substituting the results in Eqs. l|66|l and (|68|l into Eq. I|64a|) . we obtain the expression for the mixed 
components of the polarization tensor 



iC(o) 



4 v / aaJ/i 2 
9V3tt 



Sfj,6(Sfx — A) 
VV " A 2 



(69) 



It is important to emphasize that the explicit result for the electrical 00-component of this mixing gluon-photon 
polarization tensor is vanishing at po = and p — > 0. This is similar to the ideal 2SC case considered in Ref. |30j . 
(Because of this, one should be careful when interpreting the results for the Debye screening masses in a different 
basis of gauge fields H3-) The expressions for the Debye screening masses read 



l D,8S 



4a s n 2 



8ap 2 ( | 3fytfl(<fya-A) 
3tt i 



(70a) 
(70b) 



In order to obtain the Meissner screening masses, we first derive all components of the polarization tensor that span 
the space of the 8th gluon and the photon. At po = and p — > 0, the corresponding nonzero components, denoted as 
m M,AB' read 



l M,88 



m 



M,8-y 



Aa s p 2 
9?r 

27tt I 

AJaa s Ji 2 



9V3tt 



1 - 



5 fj, 9(8(1 - A) \ 
S(i9(Sp- A) \ 

s^e(Sp-A) 

k ~ V(5m) 2 - A* 



(71a) 
(71b) 
(71c) 



14 



and m 2 M 78 = m 2 M 8 . The mixing disappears in the basis of the fields in Eq. I|53[l . The Meissner screening mass for 
the 8th gluon field is 

2 _ 4(3a s + a)fl 2 f _ 5^6(5^- A) \ 

M > m ~ 27ir y- v /(^) 2 - A 2 J ' 1 j 

and the Meissner screening mass for the medium photon 7 is vanishing, m 2 M - - = 0. This is consistent with the 
absence of the Meissner effect for the unbroken U(l) cm . 

As is easy to see from Eq. <|72|) . the medium modified 8th gluon has a chromomagnetic instability in the gapless 
2SC phase. This is because the Meissner screening mass squared is negative when < A/Sfi < 1. 

VII. GLUONS WITH A = 4, 5, 6, 7 



After performing the traces over the color, the flavor and the Nambu-Gorkov indices, the diagonal components of 
the polarization tensor (P) with A = B = 4, 5, 6, 7 have the form 

n 2 T ( H 3 \( 

Kl(P) = Vy J^r^b"Gt(K)^Gt(K')+ rGt(Kh"G+ (A") 
+ 7^3 ( A) 7 'Gr (A') + 7 ^r (A) 7 "G 3 - (A') 

+ 7 M G 4 - ( A) 7 'G 2 - (A') + <fC% (K)*fGj (A')] . (73) 

[Note that II 44 (P) = II55 (P) = Ilgg (P) = TL^(P).] Apart from the diagonal elements, there are also nonzero 
off-diagonal elements, 

iC(p) = -ngT(P) = n^(p) = -n^(p) = ;n^(p), (74) 

with 

3 2 T y d 3 k 

(2^)3 

+ 7 M G+(A) 7 "G+(A') - 7^G+(A) 7 ^G+(A') 

- f G3 ( A) 7 ^r (A') + 7 "G7 (K)-fGj (A') 

- {K)^G- 2 (A') + 7 ^G 2 - ( A) 7 "G 4 (A')] , (75) 

The off-diagonal components of the gluon self-energy in Eq. I|74ll arc nonzero in general. The physical gluon fields 
in the 2SC/g2SC phase are the following linear combinations: A 4 ' 5 = (A% ± iAg)/y/2 and A% 7 = (A% ± iA%)/y/2 
|29|. These new fields, A% 6 and ^75, describe two pairs of massive vector particles with well defined electomagnetic 
charges, Q = ±1. The components of the polarization tensor in the new basis read 

n^(p) = n^(p) = n^(p) + n^(p) 

ff 2 T y d 3 k 



and 



n^(P) = 9 —J —^^[^Gt{K)^Gi{K')-rGt{K)^Gi{K') 



, / !2T)3 -Tr D [7^G3 + (A)7^G+(A')+7 M Gr(^)7 1/ G 3 -(A') 
+7 AI G+(A) 7 -G+(A') +7^G 2 -(A)7^G 4 -(A')], (76a) 

n^(p) = n^(p) = n^(P) - n^(P) 

= ^ J -^Tr D [YG+(K)rGi(K')+rGs(K)^(K') 

+rG+{K)l v G+(K') +-fGl (K)YG^(K')]. (76b) 
After Matsubara frequency summation, the polarization tensors can be written in the following form: 

ft^(P) = n a s J -j^ [C?+T£ + CitT^_ + Cf_T» v _ + Ct\T^] , (77a) 

[Cf + T^ v + + c 5 J_T^_ + Cf_T» v _ + C 5 _ 5 + T»Z] . (77b) 

(Z7T) 
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The explicit expressions for the coefficient functions C ± 4 _J. 55 and C ± ^ 55 at zero and nonzero temperatures are given in 
Appendix IB 21 Here we quote only the approximate zero temperature results at p = and p — ► 0. The particle- hole 
contributions are 



44,55 
±± 



1 E A.k - E k 



E A,k E A,k 



E, 



[8(- E bu,k) 



(-E. 



E 



b,k 



El 



b,k 



~ E A,k + Sri {E^ k f + E~E- k 

E A,k ( E A,k) 2 (W ' 



and the particle-antiparticle contributions are 



C 



44.55 



-2 



\k — fi\ -\- k — fi 



\k - fi\(k + fi b + \k - 



(78) 



(79) 



where E^, = E^ ± pb & n d pb = P — Ms- The vacuum contribution was subtracted in Eq. I|79|l in order to remove 
ultraviolet divergences in the polarization tensor. 

To proceed, we need to calculate the following two types of momentum integrals (for the details of the calculation 
see Appendix IC 2|) : 



/ 



dkk 2 Ci\ 55 



-2p 2 
-2p 2 



A 2 (A 2 + fi 2 ) 2 -4(fi 8 6nf A 2 A 4 - fi 2 1 6 ii ~ t/SJj? - A 2 

1 ~ In ——. 77 In - 

4 M 2 A 4 4 M 2 



2A 2 " A 2 



A 4 - M 2 (fy + 
, for fi 8 -> 0, 



r^(^-A) 



and 



dfcfc 2 C^ 55 



(80a) 



(80b) 



By substituting the results in Eqs. Ij48(l and 180(1 into Eq. 1(771) . we derive expressions for the polarization tensors 
in the static long- wavelength limit: 



li 44,55 



n^(o)~n^(o) 



2a s 
3tt 



4a s ^ 2 



a s p 2 A 2 
3tt ^ 2 



Afc 2 [3C 44 + u' 1 u 1 ' - (C 44 + + 2C 44 _) {cT - u^u")] 

A 2 (A 2 + /i 2 ) 2 -4(/i 8< 5 Ai ) 2 A 2 A 4 - fi 2 (Sfi - V V - A 2 
1 r—n In —-, : — - In 

t4 (sfi + v^p 



4//. 



A 4 



4 /»i A 4 -- 2 



A- 



•(«)>- A) 



, (A 2 + M 2 ) 2 -4(^) 2 A 
in ——. h In 



4 - M 2 (^-W-a 2 )' 



A 4 



A 4 



9 (Sfj, - A) 



- u'V) . 



(81) 



In the neutral 2SC/g2SC phase of quark matter, the value of the color chemical potential fi$ is small [l4|. Therefore, 
a good approximation for the above polarization tensors is obtained by taking the limit fi 8 — > 0, 



fi£(0)=fi£(0) 



2a s fi 2 



2a s fi 2 
3tt 



A 2 + 2(Sn) 2 _ _ 2 l»y/SjT 2 



A 2 A 2 

A 2 -2(^) 2 <W^ 2 ~A 2 
A 2 A 2 



9 (Sfj, - A) 
9 (Sfi - A) 



Therefore, the corresponding Debye and Meissner screening masses are 

2 2a s p 2 



Win 44 



rn 



M,44 



'13,55 



l M.55 



2q ; ,^ 2 
3tt 



A 2 -2(^) 2 , o ^v^ 



A 2 



A 2 



-0(8fi-A) 



(82) 

(83a) 
(83b) 
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11), (22), (33) 



, 3St * 0(8u - A) 



44), (55), (66), (77) 



+ 



>(<5/x-A) 



3 ' \A*^) - A2 



87), (78) 



<5m 9(<5/j-A) 
V<*M) 2 -A2 



77) 



2q / 1 1 3ap9((?M-A) 



V(«M) 2 -A2 



3a s +ck 



1 + ■ 



1 + 



3<Sm A(6m-A) 
2^/(«m)2-A2 



3a, + a f _ <5p fl(<S^-A) 



V(^) 2 -A- 



87), (7§) 



3a s +a 



3<5m gfju-Aj 



|SL. ( 1+ 



77) 



3a s + a 



_£m_£(£/£-A)_ 

V(<5m) 2 -A2 



TABLE I: The Debye and Meissner screening masses for gauge bosons in neutral two-flavor quark matter. By definition, 



m g = <iasfJ. /3ir. 



It is clear from Eq. (|83b[) that the corresponding four gluons have chromomagnetic (plasma type) instabilities in the 
gapless 2SC phase (0 < A/Sfx < 1), as well as in the gapped 2SC phase when 1 < A/Sfi < \/2. This is indicated by a 
negative value of the Meissner screening mass squared. 



VIII. DISCUSSION 



The results of our calculation for the screening masses in neutral two- flavor quark matter are summarized in TableQ] 
There we list the squared values of the Debye and Meissner screening masses in units of m 2 = Aa s p, 2 /3-K. 



A. Debye screening masses 



Let us first discuss the results for the Debye screening masses. The values of m 2 D x /m 2 with x — (11), (44), 
as functions of the dimensionless ratio A/ 6fi are shown graphically in Fig. ^ (we do not show the Debye screening 
mass for the photon whose value is suppressed by the fine structure constant). The vanishing value of the ratio A/Sfx 
corresponds to the normal phase of quark matter. In this limit, as is easy to check, our results for the Debye screening 
masses coincide with the known results in Re fs. 14*3, 145 $ . Also, in the other limiting case, A/Sfi = 00, our results 
coincide with those in the ideal 2SC phase [""""", l3Cl||. As before, by the ideal 2SC phase of quark matter, we mean the 
2SC phase without any mismatch between the Fermi momenta of the up and down quarks, i.e., 5fi = 0. 

We see that the value of the Debye screening mass for the gluons of the unbroken SU(2) C gauge subgroup is 
vanishing in the 2SC phase (A/6fi > 1). This result can be understood in the same way as in the ideal 2SC case |29j . 
It reflects the fact that there are no gapless quasiparticles charged with respect to the SU(2) C subgroup in the ground 
state. In the g2SC phase, on the other hand, there are gapless charged quasiparticles around the two "effective" Fermi 
momenta p°p = /j 



fl ± \J (S/i) 2 — A 2 [n], [lj| • The densities of states at the corresponding Fermi surfaces are 



dN 
~dE 



±\2 



g^ ± ) 



(84) 



As one can check from Table H] the squared value of the Debye screening mass m 2 D n is proportional to the sum of 
these densities of the gapless states (up to higher order corrections). Turning these arguments around, one could 
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FIG. 1: Squared values of the Debye screening masses as 
functions of the dimensionless ratio A/Sfj, for the gluons 
with A = 1,2,3 (solid line), for the gluons with A = 
4, 5, 6, 7 (long-dashed line), and for the 8th gluon (short- 
dashed line). By definition, m 2 = 4a B p- 2 /Sir. 



FIG. 2: Squared values of the Meissner screening masses 
as functions of the dimensionless ratio A/8fi for the glu- 
ons with A = 1,2,3 (solid line), for the gluons with 
A — 4,5,6,7 (long-dashed line), and for the 8th gluon 
(short-dashed line). By definition, m 2 = Aa s fi 2 /Six . 



obtain the result for the Debye screening mass without calculating the polarization tensor. Indeed, one can write |3l| 

(85) 



dN 

m ^ ~ IE 



dN 

He 



and determine the overall coefficient a s /TT by matching the expression on the right hand side with the known expression 
in the normal phase (A/5)i = 0). 

It is interesting to consider m 2 D n in the limit when the gap A approaches S/i from the side of the gapless phase. 

As one can see, the formal value of m 2 D n goes to infinity as a s p, 2 /^/l — (A/S^j,) 2 when A — > 6fi — 0. This is the 
consequence of having a quadratic dispersion relation for gapless quasiparticles E~^_ ~ (p — /i) 2 /2A when A — * 5[i. 
However, the infinitely large value of the Debye mass has little physical meaning. This is because the distance scales, 
at which the corresponding screening is set in, also become infinite in the limit A — > d/j,. 

From Table H we see that there is no mixing between the Debye screening masses of the 8th gluon and of the 
vacuum photon. It is known, however, that the physical modes of the corresponding gauge bosons in the 2SC/g2SC 
phase are the linear combinations given in Eq. (|53[) which are different from the vacuum fields. Moreover, one of the 
linear combinations, see Eq. I)53bl) . describes the medium modified photon of the unbroken electromagnetism. One 
might think that the absence of mixing between the electric screening masses is in conflict with the gauge invariance 
of the 2SC/g2SC ground state with respect to U(l) cm . However, the two propagating modes of the low-energy photon 
7 of U(l) em have transverse polarizations and, therefore, should come from the magnetic sector. The third, electrical 
mode of 7 is not massless. This latter decouples from the low-energy theory and its presence is irrelevant for the 
gauge invariance with respect to U(l) em - 



B. Meissner screening masses 



Now we discuss the results for the Meissner screening masses. The values of m 2 M x /m 2 g with x = (11), (44), (88) 
as functions of the dimensionless ratio A/S^l are shown graphically in Fig. [5] (the screening mass for the photon, 
suppressed by the fine structure constant, is not shown). As it should be, the results in the two limiting cases, i.e., 
A/S/j, = and A/Sfi = go, coincide with the results in the normal phase (i.e., no Meissner effect) and with the results 
in the ideal 2SC phase [23, y(| , respectively. 

In agreement with the group theoretical arguments, the Meissner masses of the three gluons of the unbroken SU (2) c 
subgroup are vanishing in both the gapless and the gapped 2SC phases, see the solid line in Fig. [21 The vanishing 
Meissner masses come as a result of the exact cancellation between the diamagnetic particle-antiparticle and the 
paramagnetic particle-hole contributions. 

The results in Table [I] show that the Meissner screening masses in the subspace of the 8th gluon and the vacuum 
photon have a nontrivial mixing. As we saw in Sec. IVII the mixing disappears after switching to the description in 
terms of the physical modes, defined in Eq. (|53|) . Moreover, the Meissner screening mass of the medium photon is 
vanishing. This is consistent with the absence of the Meissner effect for the unbroken electromagnetism. 
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The most interesting results of our calculation are the expressions for the Meissner screening masses of the five 
gluons that correspond to the broken generators of the SU (3) c group. We find that the squared values of these masses 
are negative (i.e., the masses themselves are imaginary) in some regions of parameters. In particular, we obtain 
rr? Mx < for x = (44), (55), (66), (77) when < A/6fi < V2, and m^ gg < when < A/Sfj, < 1. The standard 

interpretation of such a result is the existence of a chromomagnetic (plasma type) instability in the corresponding 
phases of matter [2^ . 

While the instability connected with the 8th gluon appears only in the g2SC phase (0 < A/Sfi < 1), the instability 
connected with the other 4 gluons also develops in the gapped 2SC phase when 1 < A/8/j, < y/2. This suggests that 
the gapless superconductivity itself is not the reason for the instability. It is interesting to note, however, that the 
critical value of the mismatch, above which the instability starts to develop, is given by <5/i c = A/y2. This happens 
to coincide with the value of the mismatch at which the 2SC phase becomes metastable when the neutrality condition 
is not enforced in quark matter (i.e., when the value of 5/j, is treated as a free parameter and the Coulomb effects are 
ignored). Here, on the other hand, we consider neutral quark matter in the same way as in Refs. PH. 

It is natural to suggest that the instability, indicated by negative values of the Meissner screening masses squared, 
may result in some type of a gluon condensation. Indeed, in terms of the gluon effective potential, the appearance of 
a negative gluon mass squared could be viewed as considering a false vacuum that corresponds to a local maximum 
of the potential at (A^) = 0. In this case, the true vacuum should be given by the global minimum of the gluon 
potential. It would be natural then if the minimum corresponds to another (nonzero) expectation value of the gluon 
field, i.e., (A*) ^ for a € (4, 5, 6, 7), or for a = 8. 

Note that we do not exclude the possibility that the new stable ground state has a condensate that breaks the 
rotational symmetry of the system. In fact, this might be the most natural outcome of the gluon condensation if its 
mechanism is the same, e.g., as in Ref. |43,|4||. The possibility of breaking the rotational symmetry is also suggested 
by the fact the it is the magnetic components of gluons A? rather than the electric gluons Aq that drive the instability. 
The expectation that the rotational symmetry is broken in the ground state may also hint at the possibility of a state 
with deformed quark Fermi surfaces which was proposed in Ref. |5Clj| . 

It may happen that not only the rotational symmetry but also the translational symmetry is broken in the true 
ground state of neutral two- flavor quark matter. This would be the case when the diquark condensate is inhomogeneous 
like, for example, in the crystalline phase [5l]], or like in the Abrikosov's vortex lattice phase [S^]- Regarding the 
crystalline phase (also known as the Larkin-Ovchinnikov-Fulde-Ferrell phase [lj|) in two-flavor quark matter, it might 
be appropriate to mention that this phase is claimed to appear precisely when A/S/i < It is not clear, however, 
how a first order transition from the 2SC phase to the crystalline phase at A/S/i = \/2 can be triggered by vanishingly 
small tachyonic masses of gluons. 

We would like to emphasize here that the gluon type instability, indicated by negative values of the Meissner 
screening masses squared, has nothing to do (at least, directly) with the so-called Sarma instability 54]. It was shown 
for the first time in Ref. [Ill ], and then confirmed in Refs. 

OEQHEHEII3 that the Sarma instability 
in the effective potential for the order parameter is removed when the neutrality condition is imposed, or when the 
pairinginteraction has a specific momentum structure and the ratio of the densities of the pairing fermions is kept 
fixed |23 . One may still argue that, despite the absence of the Sarma instability in the effective potential, there exists 
another type of an instability, discussed in Ref. [55j. There it was suggested that paramagnetic currents should be 
spontaneously induced in the gapless phase. In fact, we think that the instability connected with the 8th gluon in the 
g2SC phase could indeed be of this type. 

One may find some similarity between the chromomagnetic instability found in this paper and the instabilities that 
have been discussed in Refs. |5a, |53, |53, |5£a, ull m application to a state of matter produced in heavy ion collisions. It 
should be emphasized, however, that the quark distribution functions are completely isotropic in momentum space in 
the case of dense quark matter studied in this paper which is in contrast to a typical situation in Ref. |56Ll57ll53 . l59ll60| . 
This fact suggests that the mechanism of the instability seen here is more subtle. 

In passing, we would like to mention that there might exist some analogy between the instability found in this paper 
and the so-called paramagnetic superconductivity discussed in condensed matter physics [6lJ . If these phenomena are 
related indeed, it would be natural to expect that the instability of neutral dense quark matter is resolved through 
a spontaneous chromo-magnctization. Again, the breakdown of the rotational symmetry in this case would seem 
inevitable. 



IX. CONCLUSION 

In this paper, we calculated the Debye and Meissner screening masses for the gluons and the photon in the case 
of neutral, /3-equilibrated two-flavor dense quark matter. A general form of our result allows to use it in the gapped 
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phase as well as in the gapless color superconducting phase by a simple change of the magnitude of the diquark 
pairing. This latter determines the ratio of the gap parameter and the mismatch of the quark Fermi momenta, i.e., 
A/5/i. The qualitative dependence of this ratio on the diquark coupling constant in the case of neutral quark matter 
was studied in Refs. hlLhJ. It was shown that the ground state corresponds to the normal phase at weak coupling, 
i.e., A/Sfi = 0. At some intermediate values of the diquark coupling, the ground state is the g2SC phase. The ratio 
A/S/j, is less than 1 in such a gapless phase. Finally, at large coupling, the ground state is the 2SC phase, and the 
ratio A/ 5[i is larger than 1. One could also check that this ratio A/5/i increases monotonically with increasing the 
coupling. 

The results for the Debye and the Meissner screening masses in this paper give a natural interpolation between the 
known values in the normal phase [44L l45| and in the ideal 2SC phase |29l |3(j ■ The most important result of our 
calculation is that the expressions for the Meissner screening masses of the five gluons, corresponding to the five broken 
generators of the SU (3) c group, are imaginary. This is interpreted as an indication of a chromomagnetic instability 
in the corresponding phases of quark matter. It remains to be clarified the consequences of such an instability, and 
the nature of the true ground state in neutral two-flavor quark matter. 

In the future, it would be very interesting to investigate whether a chromomagnetic instability also develops in the 
gCFL phase Eg, Ell EH j where the low-energy quasiparticle spectrum resembles the spectrum in the g2SC phase. 
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APPENDIX A: CALCULATION OF DIRAC TRACES 



Here we calculate the traces defined in Eq. 142|) . By neglecting corrections of order p 2 //i 2 , we derive the following 
results: 



2 ±± 



tr 
tr 
tr 
tr 



yy A ( ± y 7 o A (T) 



yy A ( ± y 7 5 A (±) 



yy A (±y 7 5 A (T) 



2 vPvT t 



u^k" + k^u v k^k" 



-2 ^ - u^u" + 
-2 (g^ - u^u v + 



k k 2 



k 2 
k»k v 

k 2 



k k 



(Al) 
(A2) 
(A3) 
(A4) 



where k' = k — p and = (1, 0, 0, 0). In calculation of the polarization tensor, we also encounter the following type 
angular integrals: 
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where £ is the cosine of the angle between the vectors k and p. These relations are used in Eq. I|44|l . 
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APPENDIX B: COEFFICIENT FUNCTIONS USED IN CALCULATION OF POLARIZATION TENSOR 

1. The coefficient functions for Il^f with A, B = 1, 2, 3, 8, 9 

As we saw in the main part of the paper, see Eqs. (|41|l . Hfiljl . and Ij67|l . all components of the polarization tensor 
E$? with A,B = 1,2,3,8,9 were given in terms of the following coefficient functions (here and below, a = 1 for Cl l e 
and Cei 2 e 2 > while a = -1 for Cf^ and Cf^, and e\, e 2 = ±): 
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At zero temperature, these functions take the following form (here and below, the upper sign corresponds to the 
first case, e.g., C+-+, an d the lower sign corresponds to the second case, e.g., C 22 + ): 
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2. The coefficient functions for fl^f with A, B = 4, 5, 6, 7 



The components of the polarization tensor IT^f with A,B = 4, 5, 6, 7 are given in terms of the following coefficient 
functions: 
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At zero temperature, the expressions for these coefficients become 
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E A.k> - E k' 



E A,k E A,k + E b,k> + PO E A,k> E A,k' + E bM - PO 
0(- E A,k> + ¥) { E A,k>? + E k> ( E b,k -PO) 
E A,k' (BA.k^-i^k-Po) 2 

i E7 k + E k , - po 
0(- E bu,k) + 0(-E- d J 



( E lk') 2 -( E bk'-Pof 
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E A.k + E k 



E A.k> + E k> 



E A,k E A,k + E tk> + PO E A,k> E A,k> + E tk + PO 

e(-E Ak , + Sjm) {E Ak ,f - E k ,(E+ k +p Q ) 

E A,k> " ( E AM>) 2 -(Kk+P0) 2 

9(- E A.k + Sfi) (E AM ) 2 E k (E+ k , + Po ) 
( E A. k ) 2 -i E L-+Po) 2 ' 



E 



A.k 
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E A.k + E k 



E A.k' + E k> 



E A,k E A,k + E b,k> - PO E l, k < E tk, + E b,k - PO 

E b,k ~ E k'~P0 



o(- E bu, k ) + e(- E b d , k ) 

0{- E bu,k>) + 8{- E M,k>) 



i E U>) 2 -{ E b,k-Po) 2 
E b,k>- E k-PQ 

( E t k ) 2 -( E b, k >-por 
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°++ - 



+ 



E A,k " E k 



E A.k' - E k> 



E A,k E A,k + E b.k' - PO E A,k' E A,k> + E b,k + PO 

6(-E7 k , + Sim) {E Ak ,f + E k ,(Er k +p ) 



E 



A,k' 



(E A ^r-(E- k+P() ) 



0(- E L, k )+(>(- E b - d ,k) 



E, 



b,k 
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E A.k> - E k> 



(E A , k ,r-(E^+ Po r> 
i 



E A,k - E k 



+ 



E A,k' E A,k' + E b,k ~ PO E A,k E A,k + E b,k' + PO 

9(-E Ak + frz) {E Ak f + E k {E^ k , + Po ) 

E A,k ( E A.k) 2 -( E b,k'+P0) 2 

0(- E L k ') + 0(- E b d ,k') 



E b,k'+ E k +Po 



Cf_ = - 



( E A,k) 2 -( E b,k'+P0) 27 
1 



E l.k>+ E i 



i E ik+K 

E tk E tk + E b,k' + PO E l, k > E A,k< + E b,k + Po 

E b,k~ E k'+PO 



0(- E bu,k) + 8(- E b<i,k) 

e(- E bu, k ') + e(- E b - d ,k') 

1 E A.k + E k 

E A,k E A,k + E b,k'-PO 



( E A,k>) 2 -( E b,k+P0) 2 
E b,k'- E i+PO 

( E ik) 2 -(^+po) 2: 

1 E A.k> + E k> 



+ 



+ 



E A.k> E A,k>+ E b,k-PO 

6(-E Ak , + 5rf {E Ak ,f - E k ,(E+ k - po) 

E A,k> ( E A.k>) 2 -(Kk-P0) 2 

6(-E Aik + Sri {E Ak f E k {E+ k , -po) 



E 



A,k 



( E A.k) 2 -( E b + .k>-P0) 2 ' 
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APPENDIX C: MOMENTUM INTEGRALS 

1. Polarization tensor II £f with A, B = 1, 2, 3, 8, 9 

In momentum integrals that appear in the expressions for the polarization tensor Il^f with A, B = 1, 2, 3, 8, 9, one 
can use some useful approximations that simplify the calculations considerably. For example, in all integrals that come 
from particle-hole loops, the main contribution comes from a close vicinity of the average quark Fermi momentum 
p~ /2. In these integrals, therefore, it is justified to make the following replacement: 



/>oo />oo 

/ dkk 2 (...) p _ h ~fi 2 dfc(...)p.h. 

JO JO 



(CI) 



The corrections to such momentum integrals are suppressed by inverse powers of £l. To be consistent with this 
approximation, the antiparticle-antiparticle loops should be omitted, and the dependence of the particle-antiparticle 
loops on A, Sfi and may be neglected. As a result, the only type of the particle-antiparticle contribution, that 
appears in the calculation, has the form 



r dk e f.J*±±L_ + 1 W kdk £ 

J { \k-fi\(k + fi+\k-fi\) + k)-J 2" 



(C2) 
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The particle-hole loops give rise to the integrals of the following type: 



dkk 2 



A 2 



dkk 



2 E k 



J A.k 



A 2 



o 



\^A,k> 

, A 2 
dkk 2 — — 

'0 C^A.fc) 
Juu 2 ( E A,k) 2 + ( E k) 2 , f „_ , x s 

dkk 7^~T2 S (- E A,k + 6 ^) 

\ E A,k> 



2/2 2 , 



-6(-EX k + 6») ~ 0, 



2/2 2 



<5// 
A 2 



<W(<^) 2 - A 2 



(5// 



V(^) 2 - A 2 



9(6n-A), 



y/(^) 2 - A 2 
5/j 



6(6(1 -A). 
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(C4) 
(C5) 
(C6) 

(C7) 



2. Polarization tensor IT^f with A,B=4,5,6,7 

The calculation of the polarization tensor Et[~ with A,B = 4,5,6,7 reduces to the calculation of the integral in 
Eq. JSUaJ. This can be written as a sum of two expressions. One of them comes from integrating the first line in 
Eq. (ZSJ. The result is 



E A,k( E A,k- k + Mb) 



to fc2dfc _(^A, fc ) 2 + ( fc -A)(fc-M b ) 



M»6 



A,fc 



fc 2 dfc 



£ A . fc - fc + a* 



E A,k( E A,k + fc - Mb) 

dx(2x 2 + A 2 + x^ 8 ) 



/■<5/x-/x 8 J„('o^,2 i A 2 



— Sfj,—fj,g 



(£a*) 2 -(*-A*) S 



dx(Vx 2 + A 2 +x) 



Vx 2 + A 2 (Vx 2 + A 2 - x - (i s ) 
dx(\Jx 2 + A 2 - x) "\ 



2/Z 2 



/-^-^ \/x 2 + A 2 (A 2 - 2x/i 8 - Mi) A m - M8 Vx 2 + A 2 (Va; 2 + A 2 + x + /x 8 ) 

A 2 , (A 2 +Af j) 2 -(^ 8 r 
1 --- ln A^ 
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where fif, = (j, — and x = h — (i. In this calculation, we used the following table integrals: 
dx(\/x 2 + A 2 + x) 



Vx 2 + A 2 ( v / x 2 + A 2 - x - /i 8 ) 

/" dx(2x 2 + A 2 + x/t 8 ) 
7 Vx 2 + A 2 (A 2 -2x// 8 -/i 2 ) 
dx(v x 2 + A 2 — x) 



y/x 2 + A 2 + x A 2 , %/x 2 + A 2 - x - us 

5* m : , 

Ms VI 2 + A 2 - X 

%/x 2 + A 2 A 2 , (Vx 2 + A 2 + x + ^)Wx 2 + A 2 - x) 

r- r— oln — ; ; , (CIO) 

Ms 2i4 (Vx 2 + A 2 — x — « 8 )(vi 2 + A 2 + x) 

\/x 2 + A 2 - x A 2 , Vx 2 + A 2 + x + u 8 

1 5- hi . 

/i 8 y/x 2 + A 2 + x 



Vx 2 + A 2 ( v / x 2 + A 2 + x + /i 8 ) Ms 
The other expression comes from integrating the second line in Eq. H78|) . The result is 



(C9) 



(Cll) 



,^-a) r\ 2 dk^ )2+{k - m -^ 

h- E 



(EX k r-(k-fi b )< 



= p 2 9(S(i - A) 



V^ 2 " A2 dx(2x 2 + A 2 + x// 8 ) 



-v/-5m 2 -a 2 \/x 2_ +A 2 (A 2 - 2x/i 8 - /i 2 ) 



2/*l A 4 - M |(^+V^ 2 - A 2 ) 2 
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